BEST AVAILABLE COPY 



(12) INTERNATIONAL APPLICATION PUBLISHED UNDER THE PATENT COOPERATION TREATY (PCT) 



(19) World Intellectual Property Organization 
International Bureau 

(43) International Publication Date 
22 March 2001 (22.03.2001) 




PCT 



(10) International Publication Number 

wo 01/20466 Al 



(51) International Patent Classification^: G06F 13/00, 
H04N 7/173 

(21) International Application Number: PCT/USOO/21707 

(22) International Filing Date: 8 August 2000 (08.08.2000) 

(25) Filing Language: English 

(26) Publication Language: English 

(30) Priority Data: 

09/397.492 15 September 1999 (15.09.1999) US 

(71) Applicant: HOTV INC. [US/US]; 11455 El Camino Real, 
Suite 350, San Diego. CA 92130 (US). 

(72) Inventors: SHAH,Mehul; 12633 El Camino Real #3408. 
San Diego. CA 92130 (US). SRINIVASAN, Anand; 
12718 Torrey Bluff Drive. #155, San Diego, CA 92130 
(US). SHASTRI, Vijnan; 12664 Torrey Bluff Drive. 
#216. San Diego. CA 92130 (US). 

(74) Agent: BOYS, Donald, R.; RO. Box 187, Aromas. CA 
95004 (US), 



(81) Designated States (national): AE, AG, AL, AM. AT, AU, 
AZ. BA, BB. BG, BR, BY, BZ, CA. CH. CN, CR, CU. CZ, 
DE. DK. DM. DZ, EE. ES, FI, GB, GD. GE, GH, GM, HR, 
HU, ED. IL, IN. IS, JP, KE, KG. KP, KR, KZ, LC. LK, LR, 
LS, LT, LU, LV, MA, MD, MG, MK. MN, MW, MX. MZ. 
NO. NZ, PL, PT, RO, RU, SD, SE, SG, SI, SK, SL, TJ, TM, 
TR, TT, TZ, UA, UG, UZ. VN. YU. ZA. ZW. 

(84) Designated States (regional): ARIPO patent (GH, GM, 
KE, LS. MW, MZ, SD, SL, SZ. TZ. UG, ZW). Eurasian 
patent (AM, AZ, BY, KG, KZ, MD, RU, TJ, TM), European 
patent (AT. BE, CH. CY. DE, DK, ES. R, FR, GB. GR. IE. 
IT, LU, MC, NL, PT, SE), OAPI patent (BF, BJ, CF, CG, 
CI, CM. GA, GN. GW. ML, MR, 1^. SN. TD. TG). 

Published: 

With international search report. 
— Before the expiration of the time limit for amending the 
claims and to be republished in the event of receipt of 
amendments. 

For two'letter codes and other abbreviations, refer to the "Guid- 
ance Notes on Codes and Abbreviations " appearing at the begin- 
ning of each regular issue of the PCT Gazette. 



(54) Title: METHOD AND APPARATUS FOR INTEGRATING ANIMATION IN INTERACTIVE VIDEO 




o 

(^7) Abstract: An interactive video display system includes a content provider streaming a primary video stream, an annotation 
^ data stream, and a viewing station having a video display apparatus for displaying both the primary video stream and annotation 
^ data stream. The annotation stream comprises an animated gr^hic (51) in a suitable graphic format such as (GIF). The animated 
graphic may move in the display in any direction, diiected by data in the annotation stream, and may be associated with an entity (49) 
displayed from the primary video stream and moves with the image entity. In response to user interaction with an animated graphic 
^ hyperUnk, alternative display entities may be sent in the annotation data stream. These alternative display entities may alternatively 
>^ be sent along with the primary display entities with a specification as to how they would be displayed on the basis of user interaction. 



wo 01/20466 



PCT/USOO/21707 



Method and Apparatus for Integrating Animation in Interactive Video 



Field of the Invention 

The present invention is in the field of interactive video, and pertains more 
particularly to apparatus and methods for authoring and enabling animation in 
10 interactive video user interfaces. 



Background of the Invention 

15 

With continuing development of new and better ways of delivering television and 
other video presentations to end users coupled with parallel development of computerized 
information systems such as the Intemet, there have been concerted efforts to integrate 
various systems to provide enhanced information delivery and entertainment systems. For 
20 example, developers are introducing integrated systems combining TVs with computer 
subsystems, so a TV may be used as a WEB browser, or a PC may be used for enhanced 
TV viewing. 

In some systems, computer elements such as a CPU, memory, and the like, are built 
into the familiar chassis of a TV set. In such a system, the TV screen becomes the display 

25 monitor in the computer mode. In such a system, conventional TV elements and circuitry 
are incorporated along with the computer elements, and capability is provided for a user to 
switch modes, or to view recorded or broadcast video with added computer interaction. 
One may thus, with a properly equipped system, select to view analog TV programs, digital 
TV programs, conventional cable TV, satellite TV, pay TV from various sources, video 

30 delivered on the Internet, and to browse the WWW as well, displaying WEB pages and 
interacting with on-screen fields and relational systems for jumping to related information. 
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databases, and other WEB pages. The capabilities are often integrated into a single display, 
that is, one may view a broadcast presentation and also have a window on the display for 
WEB interaction. 

In some other systems, computer elements are provided in an enclosure separate 
5 from the TV, often referred to in the art as a set-top box. Set-top box systems have an 
advantage for providers in that they may be connected to conventional television sets, so 
end users don't have to buy a new TV along with the computer elements. 

In such integrated systems, whether in a single enclosure or as set-top box systems, 
user input is typically through a hand-held device quite similar to a familiar remote 

10 controller, usually having infra-red communication with the set-top box or a receiver in the 
integrated TV. For computer modes, such as WEB browsing, a cursor is displayed on the 
TV screen, and cursor manipulation is provided by buttons or other familiar pointer 
apparatus on the remote. Select buttons are also provided in the remote to perform the 
familiar fianction of such buttons on a pointer device, like a mouse or trackball more 

1 5 familiar to computer users. 

Set-top boxes and computer-integrated TVs adapted as described above typically 
have inputs for such as a TV antenna (analog), cable TV (analog or digital), more recently 
direct-satellite TV (digital), to Internet ports, and may also connect to video cassette 
recorders and to mass storage devices such as hard disk drives and CD-ROM drives to 

20 provide a capability for uploading video data from such devices and presenting the dynamic 
result as a display on the TV screen. 

The present inventors have noted that with the coupling of computer technology 
with TV, many capabilities familiar to computer users have been made available to TV 
users. For example, ability to provide text annotation for TV presentations is considerably 

25 enhanced. Computer techniques such a Pix-on-Pix are now available, wherein separate TV 
presentations may be made in separate windows, or overlaid windows on the display 
screen. Separate windows may also support display from separate sources, such as an 
analog TV program in one window, a computer game in another, and a video conference in 
a third. 
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In typical prior art video authoring systems, end users receive a single video stream 
that contains the video data and any added annotated data such as subtitling, sponsor logos, 
information blocks, and the like. However, it is desirable to build upon the goal stated in 
the preceding paragraph above, by having separate streams, one containing video data, and 
5 the other containing annotative data, that may arrive at and end user's location via different 
delivery media and be displayed synchronously on a suitable display screen. 

In a system known to the inventors, hyper-video authoring is possible wherein a 
wide variety of added, often interactive, content is made available to viewers having an 
active connection to a video feed and an Internet connection for facilitating delivery of 

10 interactive results. The system known to the inventor uses separate authoring stations 
or systems for use in what is termed hyper- video authoring by the inventor. Hyper- 
video authoring includes creating hot spots (interactive hyperlinks) in a video, 
providing interactive regions for tracked objects (methods known to the inventor), 
inserting URL's for user interaction, providing interactive banner ads, providing 

15 interactive icons for launching instant replays of short video clips, adding audio/text 
annotation, and so on. 

Annotated content may, in some instances, be delivered via alternate paths 
(typically the Internet) and synchronized with a main video stream being viewed at the 
user's end. In other instances, added content is combined at the provider end and 

20 delivered as one stream (typically cable or digital feed) to the viewer. As a viewer 
interacts with provided content, Internet capability is utilized for the purpose of 
delivering stored information associated with the viewers interaction that may be 
viewed on the same display containing the offered video such as by a PIX on PIX 
method and so on. In some cases users, if so equipped, may elect to save and store 

25 Internet delivered content to suitable data storage apparatus. In other aspects, such 
material may be saved and stored on-line at a server adapted for the purpose. In this 
case, a viewer may at a later time retrieve the material at his or her discretion. 

One of the capabilities of the above-described system is the ability to provide 
tracking coordinates for an object in the video. By tracking the movement of an image 

30 in a video presentation, an interactive region or hot spot may be embedded and caused 
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to follow a target image in the video presentation by virtue of the tracking-coordinate 
data included in an authored steam or embedded within a main video stream. The 
interactive region may be of the form of a halo surrounding an image, a semi- 
transparent area covering an Image, a portion of the image that has been altered in 

5 some way to indicate that it is interactive, and so on. 

By clicking on the interactive portion or region associated with an image entity, 
a viewer may obtain, by virtue of Internet or WAN connection, additional information 
about the image. Such information may be delivered to a viewer in the form of a WEB 
page, an order form, text information, additional interactive links, or other conventions 

10 designed to further engage the viewer regarding the selected image. The authoring 
consists of identifying, creating, and embedding the tracking coordinates (path of the 
image) into the video data so that a subsequent author may associate a hyper-link or 
interactive region with the moving image. 

In the present invention a similar capability to the tracking and selection of 

15 image entities as hyperlinks is provided, but it not necessary that the hyperlinks follow 
moving entities in a video presentation. This new capability provides new animated 
entities in the displayed video with new and non-obvious functions. 

It is known in the art that animated graphics files are used by artists to create 
and display with animation on such as Internet Web pages. One format used for this 

20 purpose is called Graphics Interchange Format (GIF). GIF files are just one of several 
formats that may be used for animation, and are referred to here as representative and 
exemplary of all such formats. An animated GIF or other animated graphic is typically 
embedded in a web page. An animated graphic may play according to instruction 
when the web page is loaded, and others may have interactive links or animated 

25 graphics on user interaction like mouse over or mouse click. Such animations may run 
in a same physical location, or may be programmed to move across a screen or display 
either on loading or on user interaction. A GIF typically has an 8-bit construction and 
an excellent compression method termed LZW. This makes it easy to transmit GIFs 
over the bandwidth-limited Internet and to display them on a suitable screen without 
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using excessive bandwidth or memory. Other suitable graphic file formats other than 

the GIF format may also be used for such interactive content. 

Providing an executable animated graphic in a manner to be integrated with a 

primary video stream would allow authors to provide interactive graphic content on 
5 top of the video in an entertaining form without being limited to providing interaction 

capabiUty to an image already in a main video. Such interactive overlaid content may 

be used for interactive advertisements. 

What is clearly needed is a method and apparatus for embedding and 

interactively linking animated graphics into a main video or authored control stream 
10 such that the graphics would change or move on the screen or appear and disappear 

within the video automatically and/or by virtue of user participation such as mouse or 

keyboard interaction. Such a method and apparatus would expand prospects for 

interactive advertising and other information display. 



15 

Summary of the Invention 

In a preferred embodiment of the present invention a video system is provided 
comprising a content provider streaming a primary video stream and an annotation 
20 data stream; and a viewing station having a video display apparatus for displaying both 
the primary video stream and the annotation data stream. The annotation data stream 
comprises one or more animated graphics in a format such as the 
Graphics Interchange Format (GIF) which is synchronized and displayed on the video. 
In some embodiments the animated graphic moves in one or both of vertical or 
25 horizontal direction, or any other trajectories in the display, directed by data in the 

annotation data stream, and may be associated with an image entity displayed from the 
primary video stream, moving with the image entity. 

The viewing station may further comprise a user input device for cursor control 
such as a mouse, and the animated graphic may be displayed as a hyperlink. User 
30 interaction with the animated graphic may either send a back-link signal to the content 
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provider or may execute some other action as per information in the annotation data 
stream. 

If an action is specified in the annotation data stream in response to user 
interaction - a back link signal may or may not be used. Such actions on user 

5 interaction specified in the primary annotation data stream could be displaying an 
alternative graphic content, changing the motion of the graphic content, making the 
graphic content appear or disappear from the video or display other specific 
information in text or graphic format. Such content may be used for interactive 
gaming kind of applications to invite more participation firom the user while he is 

10 watching the video. The advantage of specifying data about the action to be performed 
on user interaction in the primary annotation data stream is that a back-link signal is 
not required and the information about response to user interaction can be sent 
simultaneously to all users without requiring a separate communication channel for 
each user. 

15 If the user interaction is sent to the content provider as a back-link signal, the 

content provider may, as a response, stream an alternative annotation stream, 
providing an alternative display on the video display apparatus at the viewing station. 
In this case the alternative annotation stream which contains the response to the user 
interaction is sent specifically to users who interact with the content. 

20 In some cases the viewing station comprises a first high-bandwidth dedicated 

connection for receiving the primary video stream, and an Internet connection for 
receiving the annotation video stream, and the content provider comprises a first 
facility for streaming the primary video stream via the first connection, and an Internet- 
connected server for streaming the annotation video stream via the Internet. In these 

25 systems the annotation data stream first provides a first displayable indicia as a 
hyperlink, and in response to a backlink signal to the Internet-connected server 
initiated by user interaction with the first displayable indicia, the server provides 
additional information about the action to be performed such as displaying a web page 
or displaying other graphics on the video. 



wo 01/20466 PCTAJSOO/21707 

-7- 

In some embodiments the first displayable indicia responds to a user interaction 
such as a mouse-over to display other animated graphics which in turn respond to user 
interaction to display other graphics or textual content or send a second backlink 
signal. In some of these embodiments the server responds to the second backlink 

5 signal by sending a third displayable indicia in the annotation data stream. 

In another aspect of the invention an interactive video system is provided, 
comprising a computerized display system comprising a video display apparatus, 
receiving apparatus for receiving and displaying both a primary video stream and an 
annotation stream on the display apparatus, a user input apparatus for a user to interact 

10 with display entities, and a mechanism for sending backlink signals from the 

computerized display system; and an Internet-connected server for providing at least 
the annotation data stream and connected to the computerized display system in a 
manner to receive the backlink signals. The annotation data stream provides a first 
displayable indicia as a hyperlink, and the server responds to a backlink signal resulting 

15 from user interaction with the first displayable indicia by providing in the annotation 
data stream a second displayable indicia. 

In embodiments fi'om this the second displayable indicia may also be a 
hyperlink. Also in these embodiments the first displayable hyperlink may send a first 
backlink signal as a result of a mouse-over, and the second display able indicia may 

20 then send a second backlink signal as a result of a mouse click. The second displayable 
indicia may be a company logo, which may display for a pre-determined time period 
and then disappear, and the annotation data stream from the server then provides a 
third displayable indicia. One of the first and second displayable indicia may comprise 
an animated Graphics Image Format (GIF) display. In this case the second GIF may 

25 persist in the display for a pre-determined time period, then disappear, and the 
annotation data stream from the server then provides a third displayable indicia. 

In these embodiments the first and second displayable indicia may move in one 
or both of vertical and horizontal directions in the display, directed by data in the 
annotation data stream, and may also be associated with an image entity in the primary 

30 video stream, and move with the image entity. 
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In addition to the apparatus of the invention in multiple embodiments, as 
described generally above, methods for practicing the invention using the apparatus are 
taught as well, all in enabling detail below. The functionality taught for video display 
systems according to embodiments of the invention provides new ways of advertising, 
5 and new ways of presenting information to viewers with viewer participation in 
interactive video systems, and also provides new ways of doing an e-commerce 
transaction by introducing interactive advertising . 

10 

Brief Description of the Drawing Figures 

Fig. 1 is an overview diagram of a video authoring and delivery system in an 
15 embodiment of the present invention. 

Fig. 2 is a block diagram illustrating components of the authoring station of 

Fig. 1. 

Fig. 3 A is a plan view of an interactive PC/TV illustrating display and viewer 
interaction with a first animated graphic according to an embodiment of the present 

20 invention. 

Fig. 3B is a plan view of the PC/TV of Fig. 3 A illustrating a second animated 
graphic invoked in Fig. 4A. 

Fig. 3C is a plan view of the PC/TV of Fig, 3 A illustrating viewer interaction 
with the second animated grapiiic of Fig. 3B according to an embodiment of the 
25 present invention. 

Fig. 3D is a plan view of the PC/TV of Fig. 3 A illustrating a third animated 
graphic invoked in Fig. 3C. 



30 
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Description of the Preferred Embodiments 

It is an object of the present invention to provide a method and apparatus that 
enables an author annotating a primary video stream, which may be either digital or 

5 analog, to create and provide interactive animated graphics in video presentations. In 
embodiments of the invention animated graphics, for example graphics in a GIF 
format, may be used in concert with hyperlinks of various sorts to provide activity 
sequences in an interactive video displays. The method and apparatus is described in 
enabhng detail below. 

10 Fig. 1 is a block diagram illustrating an exemplary overview of a video 

authoring and delivery system 9 as known to the Inventor and suitable for practicing 
the present invention. System 9 is exemplary of a system known to the inventor for 
receiving, authoring and sending video content to viewers who subscribe to a service 
defined by this capability. Exemplary components of system 9 include a broadcast 

15 system 1 1, a video head end 13, an authoring station 15, a wide.-area-network (WAN) 
19 having an Internet Service Provide (ISP) including modems 24 and a server 26, and 
a plurality of subscribed viewers 17a through 17n. 

Video head end 13 is defined as a source for live or pre-stored video feeds for 
the purpose of video authoring before delivery to viewers such as viewers 17a-n. 

20 Video head-end 13 delivers video feed 21, which, in this embodiment, is diverted to 
authoring station 15 for the purpose of providing annotative and interactive content 
that will ultimately be displayed on equipment maintained by viewers 17a-n. Such 
content will either be combined or muxed into feed 21 and be displayed from one 
stream delivered to viewers, or may be delivered as a separate stream to be combined 

25 or caused to display in synchronous mode on a viewer's system. 

Feed 21 may be either an analog feed or a digital feed. If feed 21 is analog, it is 
converted to digital format for the purpose of authoring. Feed 21 may continue on to 
viewers as an analog feed. In another embodiment, feed 21 may be converted to 
digital format and remain as such for delivery to viewers 17a-n. Such variations in 

30 video format, delivery methods of feed 21, and delivery methods of authored content 



wo 01/20466 PCT/USOO/21707 

-10- 

will depend, in part, on system architecture and customer equipment capability, and in 
part on the nature of added content and the intent of providers. There are many 
possible variations. 

In this example, broadcast system 1 1 is capable of video delivery in digital or 

5 analog format over a cable connection, a satellite connection, or by any other high- 
bandwidth dedicated medium. The connections illustrated between viewers 17a-n and 
broadcast station 1 1 illustrate logical connections only. As previously described, video 
feed 21 is diverted in this instance, to station 15 for authoring. 

Authoring station 1 5 is enhanced with a software application 22 which enables 

10 an author to provide video annotation to the primary video stream. The video 
annotation provided may be of a variety of sorts as described in the related prior 
application under the heading above "Cross-reference to related Documents". Such 
annotation includes, for example, associating a moving image entity (for example, a 
specific player in a sports event) in a primary video with a hyperlink by tracking the 

15 image and including coordinates of the image entity in the annotation data. 

Displayable graphics, such as icons, may be annotated to primary video streams and be 
displayed either as fixed icons or moving with tracked entities, and such icons may be 
rendered as hyperlinks, such that selecting an icon initiates a backlink signal to a 
server, just as a hyperlink works in a WEB page. 

20 In some cases in the present invention animated GIFs are provided by an author 

through software 22 to be integrated with a primary video stream and used in ways 
more fully described below. Such animated GEFs may be created at the authoring 
station or retrieved from another source, inserted as metadata in a primary stream or in 
a separate stream to be later coordinated with the primary stream, and displayed as 

25 hyperlinks at a user's facility. 

In the case of a pre-stored analog video, authoring may, of course, be 
performed off-line at the authors convenience. In this case, created content including 
animated GIFs may be timed to stream at the same time of the main video, unless, of 
course, the content is made part of the main video by virtue of VBI or HBI inserted 

30 metadata. In some cases authored content may be delivered ahead of a primary video 
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and later coordinated (synchronized) with the video. All analog video is converted to 
digital format at station 15 before authoring. Separate control streams remain digital 
and must be synchronized either at station 15, or at the viewer's end before successful 
display of an annotated presentation. 

5 Because VBIs and HBIs in analog video streams are limited in capacity for 

inserted data, and may be overwritten by downstream processes, it is preferred that 
metadata including any animated GIFs be delivered in a separate digital control stream 
to be synchronized to the associated primary video at authoring station 15, at station 
11 or at each viewer's location. 

10 A feed 21a is illustrated as output from authoring station 15. Feed 21a 

represents the analog or digital video feed 21 after leaving station 15. A control 
stream 23, illustrated as output from station 15, represents a combined control stream 
containing any GIF metadata and, perhaps other forms of annotation metadata inserted 
by one or a plurality of authors. A control stream 23 a represents stream 23 being 

15 integrated with main feed 21 for combining or synchronization at station 1 1. This 
operation may alternatively be performed within station 15, in which case, only one 
feed would be output to station 1 1 . Control stream 23 may also (and preferably) be 
delivered through WAN 19 to viewers by virtue of separate WAN connections 
maintained by viewers 17a -17n. WAN 19 may, in preferred embodiments, be the 

20 Internet. 

WAN 19 is shown, as briefly described above, having a representative server 
26 and modem bank 24 constituting an Internet Service Provider (ISP) accessed by 
users 17a-n through a public-switched telephone network represented by line 23. In 
practice separate users may have separate ISPs which may each access server 26, In 
25 embodiments of the invention software 28 at server 26 in conjunction with stored or 
otherwise accessible content provides displayable entities, including animated GIF 
entities, to users 1 7 a-n in response to backlink signals as a result of users interacting 
with hyperlinks provided by metadata integrated with or coordinated with a primary 
video stream displayed at a user's station. 
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System 9 as illustrated herein may deliver analog or digital video content to 
viewer's 17a-n along with separate digital control streams that may arrive via 2iltemate 
paths. System 9 may be altered with respect to architecture to facilitate many 
alternative embodiments as long as the main or basic components exist without 
5 departing from the spirit and scope of the present invention. 

Fig. 2 is a block diagram illustrating components of authoring station 15 of Fig. 
1 according to an embodiment of the present invention. Station 1 5 may have a 
plurality of authors 1-4 and an authoring server 27. Authors 1-4 are preferably 
operating separate computerized control stations. There may be more or fewer 

10 authors operating at station 15 without departing from the spirit and scope of the 
present invention. The inventor deems these four as adequate for the purpose of 
illustrating the present invention. Some of authors 1-4 may be automated stations and 
unmanned by live persons whereas persons assigned to specific authoring assignments 
may man some other stations. 

15 This example supports real-time authoring as demonstrated by the architecture 

of authors 1-4. Video stream 21 is fed simultaneously in to each authoring station 1-4. 
If feed 21 is analog, it is converted to digital format either before arriving to, or at the 
location of each station 1-4. Each author 1-4 outputs a control stream 25a- 25d 
respectively. Each control stream 25a-d contains metadata particular to a particular 

20 authoring assignment. For example, stream 25a contains animated GIF metadata. 

Streams 25b-d may contain other types of authored metadata. In this example, all the 
control streams are combined in an authoring server 27 and are output as one control 
stream 23, which is analogous to stream 23 of Fig. 1. 

Stream 21 re-merges after exiting control stations 1-4 (authors). In some 

25 embodiments, stream 21 is passed through server 27 for the purposed of synchronizing 
it with the combined control stream 23, however, this is not required to practice the 
present invention, as other methods for stream synchronization are known to the 
inventor and may be practiced at the provider end or at the viewers end. 

In this case, main video 2 1 and control stream 23 are separate but synchronized 

30 streams as they exit authoring station 15. In alternative embodiments, as previously 
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described, there may be only one combined stream leaving station 15. The animated 
GIF metadata is included in stream 23 with other annotation. 

In this embodiment, authoring station 1 creates and prepares annotation that 
will be associated with animated GEFs. If the authoring at station 1 includes animated 

5 GIFs then the author retrieves such GIFs from a library or other content provider, and 
inserts these as metadata into control stream 25a in real time. In many cases this first 
annotation will be a transparent hyperlink to a moving image entity in the primary 
video, as described above, or a static, displayable graphic, or icon. The initial 
authoring also includes adding interactive metadata, which provides hyperlink 

10 capability according to viewer initiation such as a mouse-over or a click or double- 
click in a viewer's interface, described in further detail below. In many cases of the 
present invention, viewer interaction with a first hyperlink provided by the authoring 
capability described provides a second display, which may be an animated GIF, 
connection to additional information, or even fiirther changes in display and action. In 

15 the case of such fiirther display resulting fi'om an initial interaction, the fiirther 

metadata annotation may provided by server 26 (Fig. 1) for example, in response to a 
backlink signal resulting from a user interacting with a hyperlink. 

User interaction with hyperlinks whether or not the hyperlinks are associated 
with animated graphics is provided by conventional methods such as through pointer 

20 operation, keyboard input or other known input techniques. 

Fig.'s 3A-3D are exemplary displays on an interactive PC/TV at a viewer's 
premise illustrating various stages of viewer interaction with animated GIFs during a 
presentation of a video. 

In Fig. 3 A, an exemplary video presentation 45 is being displayed on a viewer's 

25 PC/TV 43. For the purpose of discussion and example, the primary video is a short 
video clip sponsored by a travel agency. Presentation 45 illustrates a sequence from a 
vacation location being oflfered by the travel agency for the purpose of attracting 
vacation prospects to the area. 

As presentation 45 displays on PC/TV 43, a first animated GIF 49 appears as a 

30 dorsal fin of a fish swimming up and down a stream in the video. The path instruction 
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for GIF 49 has been programmed by an author at an authoring station into an 
annotation stream as described above, the annotation stream being sent to the user's 
station via server 26, such that GIF 49 appears to swim within the stream boundaries 
in the video sequence containing the stream. Gif 49 is provided as a hyperiink. To 

5 progress to a next level, a viewer moves a cursor to the GIF. 

There are several ways one may interact with a hyperlink. Activation may 
occur by a mouse-over, which means moving the screen cursor over the hyperiink area, 
by a click, by a double-click, or by combinations. 

In Fig. 3B the dorsal GIF 49 is terminated as a result of the mouse-over, which 

10 sends a backlink signal to server 26 (Fig. 1), which responds by sending a new 

annotation data stream to the user's station. The new annotation stream provides a 
second GIF 51 in the form of a full-bodied fish. Alternatively the GIF 51 may be sent 
along with GIF 49 in the primary annotation stream along with information about 
displaying GIF 51 as a result of user interaction to GIF 49. GIF 51 is programmed to 

15 jump in and out of the stream staying within the stream's boundaries. GIF 51 may 
continue to display throughout the video waiting on a viewer's response, or may be 
timed to terminate automatically if the viewer does not respond. In one embodiment, 
when a scene change occurs fi-om, for example, a sequence showing the stream to, 
perhaps a sequence showing a cabin offered for rent, GIF 5 1 will terminate and a new 

20 interactive and animated GIF associated with the next sequence (cabin) may 

automatically appear. If a viewer chooses, he or she may obtain additional information 
by clicking on the second GIF (51) before the scene changes. 

In Fig. 3C, the viewer has elected to click on GIF 51 in order to proceed to a 
next level. Clicking on GIF 51, in this example terminates the second GIF and 

25 executes yet a third GIF. In Fig. 3D, a third GIF 53 in the form of a full bodied fish 
executes as described above, and immediately jumps to a sidebar area or to another 
specified location in the video where upon reaching it's destination, an information 
block 55 automatically pops up. Information block 55 contains text information 
associated with the vacation spot such as accommodation availability and pricing, 
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travel directions and optimum season information, available guide information for 
fishing in the stream, or any other imaginable information that may be provided. 

In addition to simple text information, block 55 may contain interactive links in 
the form of text or graphics leading to additional sources of information such as to a 

5 WEB page hosted by the travel agency containing booking arrangements and the like. 
There are many possible arrangements. Block 55 may, in some embodiments, be an 
animated GIF with animation in the form of moving text. The use of a vacation video 
and travel agency as a host is an isolated example used simply to describe one possible 
application that may be provided in the practice of the present invention. Such an 

10 application may, perhaps, be provided in conjunction with a special interactive channel 
such as a travel channel where such interactive vacation videos regularly air. 

In a preferred embodiment of the invention authoring at authoring stations of 
authoring system 15 of Fig. 1 includes associating complete packages of icons, GIFs 
and the like with a primary video stream. For example, the sequential displays 

15 described with reference to Figs. 3 A-3C above are pre-authored as a related package 
and associated with the primary video at the same time. The alternative forms are all 
present in the annotation stream, and server 26 is progranmied to provide the first form 
until a hyperlink is activated by a first method (mouse-over), then the second form in 
response to a second signal, and so forth. 

20 It will be apparent to one with skill in the art that a very wide variety of 

desirable effects may be provided by the techniques of the present invention in different 
embodiments. For example, a first interactive hyperlink may be provided as an icon 
static anywhere in a displayed video, upon a mouse-over, the icon may show a text 
annotation, such as "click for local weather". Upon clicking, the icon may be replaced 

25 by an annotated GIF according to the weather; a blue sky with moving clouds for clear 
weather; a storm cloud with rain falling for stormy weather, and so on. 

As another example, a first annotated display may provide an icon with a text 
script "weather update". A mouseover causes the icon to change temporarily to a 
Starbucks™ logo. After a short time, the logo disappears and a weather update 

30 appears in a sidebar or other window, or by an animated GIF showing the weather, as 
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described above. In this manner, short advertisements may be provided in the process 
of performing other functions. 

In another example, a GIF may be provided in a primary video of such as a 
basketball game informing viewers that there are tickets available for an upcoming 
5 game. Clicks can be processed in a manner that the first 1 00 respondents, or even 
numbered respondents of the first 200 for example, will be eligible to buy the 100 
available tickets. The respondents identities are known, as the service is a subscription 
service, and the responses are by backlink from selecting a hyperlink. Applications to 
purchase may then be mailed to the lucky respondents, or there may be provision made 
10 for online purchase, or purchase may be automatic and billed on the subscriber's 
account. 

In yet another embodiment animated graphics may take the form of animated 
company logos or brands, and user interaction may result in any number of responses, 
including such as downloading coupons for discounts and the like. 

15 Applications may be conceived for virtually any type of video presentation 

wherein an advertiser or sponsor may gain benefit by providing graphics such as GIFs 
and information associated with them. In other embodiments, animated GIFs may be 
prepared and used purely for entertainment or educational value instead of for selling 
products or for other advertisement purposes. Therefore, the method and apparatus of 

20 the present invention should be afforded the broadest scope. The spirit and scope of 
the present invention is limited only by the claims that follow. 
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What is claimed is: 

1 . A video system comprising: 

a content provider streaming a primary video stream and an annotation data 
5 stream; and 

a viewing station having a video display apparatus for displaying both the 
primary video stream and the annotation data stream; 

wherein the annotation data stream comprises at least a first animated graphic 
which displays on the video display. 

10 

2. The system of claim 1 wherein the first animated graphic moves in the display as 
directed by data in the annotation data stream. 

3 . The system of claim 2 wherein the moving first animated graphic is associated with 
15 an image entity displayed from the primary video stream, and moves with the image 

entity. 

4. The system of claim 1 wherein the viewing station fiirther comprises a user input 
pointer device for cursor control and wherein user interaction by the pointer device 

20 with the first animated graphic causes a change in the animated graphic display. 

5. The system of claim 4 wherein user interaction with newly-appearing animated 
graphics causes additional animated graphics to appear. 

25 6. The system of claim 4 wherein user interaction with the first animated graphic 

produces a back-link signal, and in response to the backlink signal the content provider 
streams an alternative annotation stream, providing an alternative display on the video 
display apparatus at the viewing station. 
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7. The system of claim 1 wherein the viewing station comprises a first high-bandwidth 
dedicated connection for receiving the primary video stream, and an Internet 
connection for receiving the annotation video stream, and the content provider 

comprises a first facility for streaming the primary video stream via the first 
5 connection, and an Internet-connected server for streaming the annotation video 
stream via the Internet. 

8. The system of claim 7 wherein the annotation data stream first provides a first 
displayable indicia as a hyperlink, and in response to a backlink signal to the Intemet- 

10 connected server initiated by user interaction with the first displayable indicia, the 
server provides in the annotation data stream an animated graphic as a hyperlink. 

9. The system of claim 8 wherein the first displayable indicia responds to a user 
interaction such as mouse-over to send the backlink signal, and wherein the animated 

15 graphic provided in response in turn responds to user interaction to send a second 
backlink signal. 

1 0. The system of claim 9 wherein the server responds to the second backlink signal by 
sending a third displayable indicia in the annotation data stream. 

20 

11. An interactive video system comprising: 

a computerized display system comprising a video display apparatus, receiving 
apparatus for receiving and displaying both a primary video stream and an annotation 
stream on the display apparatus, a user input apparatus for a user to interact with 
25 display entities, and a mechanism for sending backlink signals from the computerized 
display system; and 

an Internet-connected server for providing at least the annotation data stream 
and connected to the computerized display system in a manner to receive the backlink 
signals; 
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wherein the annotation data stream provides a first displayable indicia as a 
hyperlink, and the server responds to a backlink signal resulting fi'om user interaction 
with the first displayable indicia by providing in the annotation data stream a second 
displayable indicia. 

5 

12. The system of claim 1 1 wherein the second displayable indicia is also a hyperlink. 

13. The system of claim 12 wherein the first displayable hyperlink sends a first 
backlink signal as a result of a mouse-over, and the second display able indicia sends a 

10 second backlink signal as a result of a mouse click. 

14. The system of claim 13 wherein the second displayable indicia is a company logo. 

15. The system of claim 14 wherein the company logo displays for a pre-determined 
15 time period followed by a third displayable indicia. 

16. The system of claim 13 wherein one of the first and second displayable indicia 
comprises an animated graphic in Graphics Image Format (GIF), 

20 17. The system of claim 13 wherein the second displayable indicia comprises a 
Graphics Image Format (GIF) display, which persists in the display for a pre- 
determined time period, then disappears, and the annotation data stream fi*om the 
server then provides a third displayable indicia. 

25 18. The system of claim 1 1 wherein one or both of the first and second displayable 
indicia move in the display, directed by data in the annotation data stream. 

19, The system of claim 1 8 wherein one or both of the first and second displayable 
indicia is associated with an image entity in the primary video stream, and moves with 
30 the image entity. 
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20. A method for providing annotated data entities in a displayed video, comprising 
steps of: 

(a) streaming a primary video stream and an annotation data stream by a 
5 content provider; 

(b) displaying both the primary video stream and the annotation data stream at 
a viewing station connected to the content provider by at least one data link; and 

(c) providing an animated graphic which displays on the video display from the 
data in the annotation data stream. 

10 

21 . The method of claim 20 wherein, in step (c), the animated graphic moves in the 
display, directed by data in the annotation data stream. 

22. The method of claim 21 wherein the moving graphic is associated with an image 
15 entity displayed from the primary video stream, and moves with the image entity. 

23. The method of claim 20 wherein the animated graphic displays as a hyperlink, and 
comprising a further step for interacting with the hyperlink by a user input pointer 
device for cursor control, such that user interaction with the animated graphic causes a 

20 second animated graphic to display. . 

24. The method of claim 23 wherein interacting with newly appearing animated 
graphics causes additional animated graphics to display. 

25 25. The method of claim 23 comprising a further step wherein interaction with an 
animated graphic causes a back-link signal, and, in response to the backlink signal, the 
content provider streams an alternative annotation stream, providing an alternative 
display on the video display apparatus at the viewing station. 



wo 01/20466 PCTAJSOO/21707 

-21 - 

26. The method of claim 20 wherein the viewing station comprises a first high- 
bandwidth dedicated connection for receiving the primary video stream, and an 
Internet connection for receiving the annotation video stream, and the content provider 
comprises a first facility for streaming the primary video stream via the first 
5 connection, and an Internet-connected server for streaming the annotation video 
stream via the Internet. 



21. The method of claim 26 wherein the annotation data stream first provides a first 
displayable indicia as a hyperlink, and in response to a backlink signal to the Intemet- 
10 connected server initiated by user interaction with the first displayable indicia, the 
server provides in the annotation data stream an animated GIF as a hyperlink. 

28. The method of claim 27 wherein the first displayable indicia responds to a mouse- 
over to send the backlink signal, and wherein the animated graphic provided in 

15 response to the backlink signal responds to a mouse click to send a second backlink 
signal. 

29. The method of claim 28 wherein the server responds to the second backlink signal 
by sending a third displayable indicia in the aimotation data stream. 

20 

30. A method for providing annotated data entities in a displayed video, comprising 
steps of: 

(a) streaming a primary video stream to a computerized video display system 
by a first streaming facility; 
25 (b) streaming an annotation data stream to the computerized video display 

system by an Internet-connected server enabled to receive backlink signals; 

(c) displaying both the primary video stream and the annotation data stream at 
the computerized video display system, the annotation data stream providing a first 
displayable indicia as a hyperlink; 
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(d) responding by the Internet-connected server to a backJink signal from user 
interaction with the first hyperlink by providing in the annotation data stream a second 
dispiayable indicia. 

5 31. The method of claim 30 wherein, in step (d), the second dispiayable indicia is also 
a hyperlink. 

32. The method of claim 3 1 wherein, in step (d) the first dispiayable hyperlink sends a 
first backlink signal as a result of a mouse-over, and the second display able indicia 

10 sends a second backlink signal as a result of a mouse click. 

33. The method of claim 3 1 wherein the second dispiayable indicia is a company logo. 

34. The method of claim 33 wherein the company logo displays for a pre-determined 
15 time period and then disappears, and the annotation data stream fi-om the server then 

provides a third dispiayable indicia. 

35. The method of claim 30 wherein one of the first and second dispiayable indicia 
comprises an animated graphic in a suitable format, 

20 

36. The method of claim 32 wherein the second dispiayable indicia comprises an 
animated graphic display, which persists in the display for a pre-determined time 
period, then disappears, and the annotation data stream fi-om the server then provides a 
third dispiayable indicia. 

25 

37. The method of claim 30 wherein one or both of the first and second dispiayable 
indicia move in the display directed by data in the annotation data stream. 

38. The method of claim 37 wherein one or both of the first and second dispiayable 
30 indicia is associated with an image entity in the primary video stream, and moves with 
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the image entity. 
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